Integration of Visual Perception in Dialogue Understanding for Virtual Humans in Multi-Party interaction

While the dialogue functions of speech in two-party dialogue have been extensively studied, there has been much less work on either multi-party communication, multimodal communication, and especially vision in a multi-party face-to-face setting. In this paper we report on one such effort to apply state of the art real-time visual processing to enhance a dialogue model of multi-party communication. We are concerned with situations in which there are at least three parties involved in conversation (at least one of whom is a human participant and at least one of whom is a virtual human). We focus on the visual behaviors of head orientation, head nods and head shakes, and examine how these behaviors influence several aspects of a multi-layer dialogue model, including addressee identification, turn-taking, referent identification, social affiliation, grounding, and question answering. We describe the extensions to the dialogue model and the implemented techniques for recognizing these behaviors and their impact on the dialogue models in real time, in realistic conversational settings from people participating in dialogue with virtual humans. We present several case studies (with accompanying videos) of dialogue fragments of the virtual agents with and without the recognition of these behaviors. Future work involves detailed studies on both the context recognition rates for this task as well as overall subjective impact on user satisfaction and dialogue efficiency.